Attribute extraction

ABSTRACT

A data object submitted for storage is analyzed, and a set of values is extracted from the data object that can correspond to a set of attributes. The analysis of the data object can also identify possible new ontology terms. One or more extracted values are presented to the entity which submitted the data object for approval and feedback. This feedback can be used to characterize the data object with appropriate terms, train the extraction process for future extractions, and/or expand the set of known ontology terms.

RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, U.S. patent application Ser. No. 13/620,641, filed on Sep. 14, 2012,entitled “Feedback Enhanced Attribute Extraction,” now U.S. Pat. No.9,087,084, which claims the benefit of priority from U.S. patentapplication Ser. No. 12/011,197, filed on Jan. 23, 2008, entitled“Feedback Enhanced Attribute Extraction,” now U.S. Pat. No. 8,285,697,which claims the benefit of priority from U.S. Provisional PatentApplication No. 60/886,296, filed on Jan. 23, 2007, entitled “FeedbackEnhanced Attribute Extraction.” The disclosure of the foregoingapplications are incorporated herein by reference in their entirety.

BACKGROUND

The present disclosure relates to digital data categorization andsearching.

Computer databases now serve as storehouses for diverse types ofinformation and media including documents, images, audio files, videos,and practically any other type of information capable of being stored ina digital format. The interconnected nature of today's computingenvironment offers the capability for users to have nearly instantaccess to this information regardless of their physical location.Searching interfaces serve as gateways to the vast information stored inthese databases, but due to the tremendous amount and diverse types ofdigital data that are now accessible, searching for a broad category ofdata or performing mere keyword searches of these data stores can returnan unmanageable number of results. The particular data being sought bythe searcher can be obscured by a cumbersomely large result set,limiting the usefulness and efficiency of the search.

To assist the searcher in retrieving the sought after data, a searchinterface can offer refinement options to narrow the list of results. Inorder to maximize the usefulness of such refinement options, the optionspresented can be based on the result set of a search and/or the searchquery itself. Creating useful a list of options for presentation to auser in narrowing a result set can be accomplished manually, but theprocess can be tedious, especially for large and diverse data sets.

SUMMARY

This document describes systems and techniques that may enhance themanner in which data objects are stored and are presented to userssearching for such data objects. For example, the systems and techniquesdescribed here may involve asking a user who uploads a data object to asystem for information about the data object. Questions posed to theuser may be based on an initial analysis of the uploaded object. Forexample, an uploaded text document can be analyzed and its contentscompared to terms stored in an ontology in order to initially classifythe document. The results of the analysis can also be used, for example,to create a list of questions that can be used to solicit feedback fromthe user who uploaded the data object in order to confirm (or repudiate)the validity of one or more classifications. Classifications confirmedby the user as valid can be stored in association with the uploaded dataobject. The associated classifications can be used later, for example,by search engine users to locate the document. The feedback can be used,for example, to train the analysis process for future use and/or toupdate the ontology.

In general, one aspect of the subject matter described in thisspecification can be embodied in methods that include the actions ofreceiving a data object uploaded from a user interface device, analyzingthe uploaded data object based on an ontology to extract a plurality ofvalues having corresponding attributes, presenting a feedback request tothe user interface device, the feedback request including at least onequestion as to whether an extracted value paired with its correspondingattribute accurately characterizes the data object, receive a responseto the feedback request from the user interface device, and store aconfirmed attribute-value pair in association with the uploaded dataobject.

Further optional features include the following features. A systemimplementing the method can train an extraction process based on aconfirmed attribute-value pair. A new term can be added to the ontologybased on a confirmed attribute-value pair. An uploaded data object canbe served to a remote computer over a network, and the data object canbe served in response to a request for data objects characterized by aconfirmed attribute-value pair. Uploaded data objects can include text,audio, image, and video files. Analysis of audio files can includespeech-to-text conversion. Analysis of image files can include an imagerecognition process. Analysis of video files can include speech-to-textconversion and an image recognition process. A candidate ontology termextracted from a data object that is not found in the ontology can beadded to the ontology if a feedback response indicates that the term iscontextually relevant to the data object.

Other embodiments of this aspect include corresponding systems,apparatus, and computer program products.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a conceptual diagram showing a process by which a user mayprovide content to a central server system.

FIG. 1B is a block diagram of an example attribute extraction system.

FIG. 1C is a schematic diagram of a system for annotating objects suchas documents and other electronic files.

FIG. 2 is a flowchart of an example method for performing attributeextraction.

FIG. 3 is an example text document data object.

FIG. 4 shows example extraction results for a text document data object.

FIG. 5 shows an example feedback form.

FIG. 6 is a flowchart of an example method for processing receivedfeedback.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1A is a conceptual diagram showing a process 100 by which a usermay provide content to a central server system 105. In general, theprocess involves receiving an object, such as an electronic file ofinformation (e.g., a database file, word processing document, web page,etc.) at the server system 105, analyzing the receive object (orobjects) and generating one or more contextual clarifying questionsbased on the content of the object, and then presenting the questionsback to a user who submitted the object. The answers to such questionsmay be used to better classify the object so that it can be located morereadily by the user or by other users seeking such an object.

The process 100 is generally exemplified here by displays 101-103 andlettered arrows between those displays 101-103 and server system 105. Inthis example, the displays 101-103 are examples of screen shots that maybe presented to a user who has logged onto the serer system 105 using aweb browser. The user may interact with the server system 105 inconventional manners to provide objects and additional information tothe server system 105. Although a single server is shown to representthe server system 105 for purposes of clarity, it will be understoodthat the server system 105 may include a number of servers performing avariety of function.

Display 101 shows that the user has already logged onto the serversystem 105, and is particularly using an application that permits theuser to upload objects to the server system 105. The particularapplication may be associated with a more general application orapplications such as the GOOGLE BASE system. In this example, the userhas already indicated a desire to upload an object, and display 101shows a pop up navigation window by which the user identifies the fileto upload.

Arrow A shows the passing of the file from the user's computer, which isgenerally located remote from the server system 105, to the serversystem 105. Upon receiving the file, the server system 105 may analyzethe file using techniques described more fully below. The analysis maypermit the server system 105 to obtain a preliminary understanding ofthe file. For example, the analysis may permit the server system 105 toidentify a title of the file, various topics addressed in the file, thepurpose of the file (e.g., is it a database of product process, aspreadsheet showing a person's schedule, a manuscript for a book, etc.),and other such information.

In normal operation, the user could simply be asked to provide suchinformation when he or she uploads the file. However, such a requestfrom the user may be burdensome, so that the user does not comply, andsimply uploads the file without responding to the request for additionalmeta data about the file. Alternatively, the user might simply enter oneor two topics—either because the user does not want to be bothered andwants to enter the minimum amount needed to make the upload work, orbecause the user does not remember all of the various topics associatedwith the object.

As shown here by Arrow B and display 102, the server system 105, throughits analysis has identified preliminary information that it providesback to the user for confirmation. In this example, the server system105 has identified a potential title of the document in a file uploadedby the user, along with two potential topics for the document. Thedisplay 102 provides a simple mechanism by which the user can confirmwhether the preliminary classifications made by the server system 105are correct. Thus, for example, the user can confirm the title inferredby the server system 105 by checking a first box, or may provide adifferent title and check a second box. Likewise, the user can confirmor refute the two suggested topics for the document that were inferredby the server system 105.

Arrow C shows the transmission of the user's selections (e.g., in anHTTP request) to the sever system 105. Upon receiving such information,the server system 105 may store it as meta data associated with theuploaded object and may also index such meta data so that it can be usedin generating results for future search queries received by the serversystem 105. Arrow D and display 103 then show a confirmation sent to theuser by the server system 105 indicating that the object has beenuploaded and classified adequately.

FIG. 1B is a block diagram of an example attribute extraction system108. The system 108 includes at least one server 109 in communicationwith one or more clients 115 via at least one network 110. The clients115 can each include a device, such as a personal computer, a wirelesstelephone, a personal digital assistant (PDA), a laptop computer, oranother type of computation or communication device, a thread or processrunning on one of these devices, and/or an object executable by one ofthese devices. The server 109 can include one or more servers thatgather, store, process, maintain, and/or manage web page impressions,and process complex queries. The server 109 can also include one or moreservers that accept uploaded data from a client 115 for storage.

Although two clients 115 and a single server 109 are shown in FIG. 1,there can be more servers and more or fewer clients. For instance, someof the functions performed by the server 109 can be performed by one ormore other servers such that the server 109 can represent severaldevices, such as a network of computer processors and/or servers. Thenetwork(s) 110 can include one or more local area networks (LANs), widearea networks (WANs), telephone networks, such as the Public SwitchedTelephone Network (PSTN), intranets, the Internet, and/or or other typeof network. The client(s) 115 and server 109 can connect to thenetwork(s) 110 via wired, wireless, or optical or other connections. Insome implementations, one or more of the devices illustrated in FIG. 1are directly connected to another one of the devices. For example, insome implementations, the clients 115 are directly connected to theserver 109.

The server 109 generally includes a processor 130, an operating system135, a memory 120 including a feedback module 125 and an attributeextraction module 126, an ontology database 140, a data objects database141, one or more interface(s) 146, one or more storage device(s) 148,and a bus 144. The bus 144 can include one or more paths that permitcommunication among the components of server 109.

The processor 130 includes any appropriate type of conventionalprocessor, microprocessor or processing logic that interprets andexecutes instructions, and works in conjunction with the operatingsystem 135 to execute instructions stored in the memory 120 and/orstorage devices 148 of the server 109. The memory 120 can include arandom access memory (RAM) or another type of dynamic storage devicethat stores information and instructions for execution by the processor130. The storage device(s) 148 can include a conventional ROM device oranother type of static storage device that stores static information andinstructions for use by the processor 130. Additionally, the storagedevice(s) 148 can include a magnetic and/or optical recording medium andits corresponding drive. According to an implementation, although theoperating system 135 is shown as separate from the memory 120 andstorage device(s) 148, the operating system 135 may be stored within thememory 120 and/or storage device(s) 148.

The server 109 includes one or more interfaces 146 that permit input tothe server 109 via one or more conventional mechanisms, such as akeyboard, a mouse, a pen, voice recognition and/or biometric mechanisms,etc. The interface(s) 146 can also permit output from the server 109 viaone or more conventional mechanisms, such as a display, a printer, aspeaker, etc. The interface(s) 146 can further include one or morecommunication interfaces that enable the server 109 to communicate withother devices and/or systems. For example, the interface(s) 146 caninclude mechanisms for permitting the server 109 to communicate with theclients 115 via one or more networks, such as the network(s) 110. Theinterface(s) 146 can permit the server 109 to communicate with otherservers, including Internet servers.

In operation, the server 109 can store data objects in the data objectsdatabase 141. In some implementations, the server is operable to receivedata objects from clients 115 for storage in the database 141. Theserver is further operable to analyze the data object stored in the dataobject database 141 to extract attributes for the data object and tosolicit feedback from one or more users regarding the results of theextraction process. In some implementations, the server 109 performsthese operations in response to the processor 130 executing softwareinstructions contained in a computer-readable medium, such as the memory120. In some implementations, the software instructions for extractingattributes from the data object are included in an attribute extractionmodule 126 within the memory 120. The attribute extraction module 126can analyze the data object stored in the data object database toextract related attributes from the data. In some implementations, thesoftware instructions for soliciting feedback regarding the results ofthe extraction process are included in a feedback module 125 within thememory 120. The feedback module 125 can present inquiries to users ofclients 115 to determine if the results of the extraction processaccurately represent the content of the analyzed data object. Thefeedback module 125 can be further configured to accept responses toadjust the results of the extraction process according to the responses,if needed, and to train the extraction process for use in laterextractions.

The extracted attributes, following any needed adjustments according tothe accepted responses, can be stored along with the data object in thedata object database 141. Alternatively, or in addition, the adjustedattributes can be stored in a separate data structure (not shown) whichincludes a reference matching the adjusted attributes to theircorresponding data object. The data objects and the adjusted attributesare preferably stored in a data structure that permits complex queriesto be answered quickly, and which optimizes the space required forstoring such data.

The software instructions can be read into the memory 120 from anothercomputer readable medium, such as the storage device(s) 148, or fromanother device via the interface(s) 146. The software instructionscontained in the memory 120 cause processor 130 to perform processesdescribed in this disclosure. Alternatively, hardwired circuitry can beused in place of or in combination with software instructions toimplement processes consistent with the disclosure. Thus,implementations are not limited to any specific combination of hardwarecircuitry and software.

The client device(s) 115 include a processor 150, an operating system155, a memory 160, one or more interface(s) 146, one or more display(s)180 one or more storage device(s) 185, and a bus 170. The bus 170includes one or more paths, such as data and address bus lines, tofacilitate communication between the processor 150, the operating system155, and the other components within the client 115. The processor 150executes the operating system 155, and together the processor 150 andoperating system 155 are operable to execute functions implemented bythe client 115, including software instructions contained in acomputer-readable medium stored in the memory 160.

The memory 160 can include random access memory, read-only memory, ahard disk drive, a floppy disk drive, a DVD, CD-ROM, or other drivetype, for storing information on various computer-readable media, suchas a hard disk, a removable magnetic disk, or a DVD or CD-ROM disk.Additionally, the interface(s) can control input/output devices of theclient 115, such as a video display, a keyboard, a scanner, a mouse orjoystick or other input or output devices. The interface(s) can alsoinclude one or more input/output ports and/or one or more networkinterfaces that permit the client 115 to receive and transmitinformation, such as from and to the server 109, such as via thenetwork(s) 110.

The server 109 and clients 115 illustrated in FIG. 1 supportcombinations of means for performing the specified functions describedherein. As noted above, it will also be understood that each block ofthe block diagrams, and combinations of blocks in the block diagrams,can be implemented by special purpose hardware-based computer systemsthat perform the specified functions or steps, or combinations ofspecial purpose hardware and computer instructions. Further, the server109 and clients 115 can each be embodied as a data processing system ora computer program product on a computer-readable storage medium havingcomputer-readable program code means embodied in the storage medium. Anysuitable computer-readable storage medium may be utilized including harddisks, CD-ROMs, DVDs, optical storage devices, or magnetic storagedevices. Accordingly, the server 109 and/or clients 115 may take theform of entirely hardware, an entirely software or a combination ofsoftware and hardware, such as firmware. Furthermore, though illustratedindividually in FIG. 1, each component of the server 109 and clients115, respectively, may be combined with other components within theserver 109 and/or clients 115 to effect the functions described herein.

The server 109 includes an ontology 140 for use in analyzing the dataobjects and extracting attributes therefrom. An “ontology” as usedherein is a data model representing terms that are relevant to an areaof interest or domain. Ontologies can describe specific objects, classesof objects, attributes of objects, and relationships between objects.Classes are sets, collections, or types of objects. Attributes areproperties, features, and characteristics of objects. Relationships areways in which objects can be related to one another.

Using the example of a word processing document, for example, a termpaper on Hawaii, the specific object is the file itself. An example of aclass into which this object can fall are “term papers” which wouldinclude all term papers within the domain, and “word processingdocuments”. Examples of attributes are the “title of the document”, the“topic”, the “type of file”, and the “length of the file”. Classes canbe described using attributes depending on the choice of attribute. Theclass of “term papers”, for example, can be described using a “type ofdocument” attribute, and the class of “word processing documents” can bealso be described using the same attribute. A class refers to allobjects in the domain that belong to the class, and an attributedescribes a characteristic of an individual object. Attributes havevalues associated with them. These values vary depending on the specificattribute considered. For example, the attribute “color” can have valuesof “red”, “blue”, “yellow”, “green”, etc. In some cases attributes aremore complex. For example, the attribute “topic” for a given object suchas an Internet blog site about a trip to Spain can have multiple valuessuch as “travel”, “Europe”, and “Spain”. An attribute along with a valueare sometimes referred to as an attribute-value pair.

A term included in the ontology is referred to herein as an ontologyterm. An ontology term can include any term used in the data model,including classes, attributes, all corresponding values for eachattribute, and relationships. Ontology terms are stored in the ontologydatabase 140.

In the description that follows, an implementation involving theextraction of attributes and/or attribute values is described, but thedescribed methods and system can be used similarly with respect to classand relationship determinations. Data objects as described hereininclude text files such as those comprising ASCII text, binary filessuch as audio data, image data, video data, and the like, and anycombination of the aforementioned data types.

FIG. 1C is a schematic diagram of a system 190 for annotating objectssuch as documents and other electronic files. In general, the system 190operates on various forms of data objects, for example, features anddocuments from different corpora 195, to assign various meta data to theobjects in the form of various attribute value pairs. Through such atechnique, the meta data can be added to objects to make the objectseasier to identify during information retrieval, for example, such aswhen users search the corpora for objects of interest by providing queryterms. The various corpora may take a wide variety of forms, such asgeneral internet web pages, local web pages (e.g., pages addressed by alocal search system), databases such as GOOGLE BASE, product search datasubmitted by product vendors or otherwise obtained, and otherunstructured, semi-structured, or structured databases of information.

Two flows of information are shown in the example. The first flow, shownby solid lines, generally involves an analysis process that can occuroff-line and can be performed intermittently or periodically, forexample, when the corpora 195 is subject to low traffic. The processinvolves taking objects from the corpora and applying one or moreanalyzers to the objects to compute a domain or domain model for theobjects, and then storing the domain models that are so derived in thedatastore 197. The other flow, shown by dashed lines, generally involvesclassification of objects or items in the corpora so as to provideannotations to the items. Such a process generally happens in real timeor near real time, for example, in response to a new item being added tothe corpora. Each of the processes includes unsupervised and supervisedcomponents. The supervised components involve the use of a questionanswerer 192 to confirm or refute suggested values for variousattributes for an object, as shown in examples above and below. Forexample, a user who uploads an item (or another user or users) can bepresented with information inferred by the system 190, and the user canprovide feedback about whether the information is accurate or not. Suchinformation fed back by the users can then be incorporated into theannotations that are stored with or related with the items in thecorpora 195.

Referring now in more detail to the process represented by the solidlines, in this example, documents are first accessed from the corpora.The documents can be, for example, web pages that have been crawled foruse in a search engine and may be formatted in a variety of manners,such as in an HTML format. The extracted documents can then be providedto one or more analyzers, which act to compute a domain model for eachdocument or set of documents. To do so, the extractors extract varioussignals from the documents and combine such extracted signals into aclass specification or domain. A domain may be considered to be similarto an object oriented programming class, in that it can have a type andvarious attributes. The domains may also be organized hierarchically.For example, a “trumpet” may be considered a child of a “brassinstrument,” which may in turn be a child of “musical instrument.” Thetrumpet may have an attribute such as the number of valves, and theattribute may have values such as three (most trumpets) or four (perhapssome trumpets).

Various forms of analyzers can also be used to conduct such analysis.For example, one analyzer can be established to analyze particular,well-understood web sites that are generally small in number, so as toproduce accurate information that has a very good correlation toparticular attributes, but has potentially incomplete information. Suchan analyzer may operate according to generally known techniques, such asby having a human operator establish parameters for a patternrecognizer, either manually or semi-automatically, and then parsing theobjects using the pattern recognizer. Using the example above, forinstance, various web sites relating to music may be patterned, and theanalyzer may identify values for various parameters, such as identifyinga trumpet that is identified as having three valves. The connectionbetween the trumpet and the three valves, or between the attribute“valves” and the value of “three”, can be relatively strong in such asituation. However, because the analysis of the site involves somemanual operation, the amount of information that can be analyzed in thismanner may be limited as a practical matter. Thus, in this example, theanalyzer only finds the value of “three” for valves, and does not “know”that there are four-valve trumpets.

Another type of analyzer can be implemented to be less precise withrespect to matching attributes to values, but much more complete in itscoverage. For example, such an analyzer may analyze a broad array of websites or other forms of documents, and may look for repeated structuresin the pages, e.g., by analyzing DOM tree organizations of web pages. Inone example, the analyzer may use parallel HTML structures such as list,table, and form structures, to find common sets in the various pages.The sets may then be expanded using overlapping co-occurrence betweenthe pages, and may also be expanded using ontology enumerations. Such anapproach may provide a weaker connection between a particular attributeand particular values for the attribute, but it may provide morecomplete value coverage because of its broader reach across a greaterarray of sites. Using the trumpet example again, such an analyzer maysee many documents that speak of trumpets, valves, and the number threein similar ways, and may also see a smaller number of documents (butstill a significant number) that speak of trumpets, valves, and thenumber four (or the numbers three and four). Thus, although the analyzermay not be able to draw as confident a link between the numbers and theterms “trumpet” or “valve”, it does have a more complete set of values.

The operation of the various forms of analyzers may also be combined, asappropriate, to improve the analysis function. In particular, there maybe overlap between the signals extracted using one extractor and thoseusing another. Where the overlap is sufficient, the attributes and/orvalues from one extractor may be used to improve the results achieved bythe other extractor. Referring again to the trumpet example, the twoanalyzers may overlap in their use of the terms trumpet, valve, andnumbers together. As a result, the more specific analyzer may use theinformation from the more general analyzer that ties the attributevalues of “three” and “four” together, and may thus extend its knowledgeto include an understanding that trumpets can have three valves and fourvalves.

When an appropriate domain model is determined for a document or groupof documents, the domain model may be provided for storage in datastore197. The analyzers may also attach a provenenance to created domainsbefore providing them to the datastore 197. The provenance is a pointerof an appropriate form back to the originating object in the corpora195. In such a manner, if information needs to be removed from thedatastore 197 or otherwise tracked back from the datastore 197 to itsoriginating object or objects in the corpora 195, such tracking mayoccur without having to affect large swaths of data in the datastore197.

Supervision may be applied to the analysis using the question answerer192. In particular, where the analyzer cannot make definitivedeterminations about an attribute, it may present the attribute to ahuman operator who may confirm or refute the information. Using thetrumpet example again, the correlation between the results of the twoanalyzers may not be complete or certain enough so that the system 190can determine with sufficient confidence that trumpets can have three orfour valves (and no other values). In such a situation, the analyzer maypose a question to a human user, such as by asking the user the numberof valves that a trumpet can have, and presenting the numbers three andfour next to check boxes (along with a control that lets the user addadditional values). Such feedback from the user may then be provided tothe analyzers 194 or the datastore 197 to update the range of theattribute values by the new value that was learned, or to confirm thealready-stored values (e.g., by adding a higher confidence level to theinformation). Such feedback may also be held back until multiple peoplehave provided the same or similar answers. For example, one user may nothave ever seen a four-valve trumpet, while three others have. By waitingfor answers from multiple users, the information that updates thedatastore 197 may be more complete and accurate.

The particular questions that are posed to a user may be chosen in avariety of manners, such as using heuristic rules. For example,particular questions or kinds of questions may be triggered when aparticular signal is identified in an item. For example, itemsidentified as probable titles of documents may trigger a certainquestion via one heuristic, while potential topics in the document maytrigger others. Various other appropriate techniques may also be usedfor generating questions or a series of questions. In general, thequestion may be selected to minimize the level of effort required by theuser to respond to the questions.

The other side of the system 190 centers around one or more classifiers196. The classifiers may take various well-known forms, including aNaive Bayes classifier or other such trainable classifiers. Theclassifiers 196 may draw objects or items from the corpora 195 and mayannotate the items using information from domains that are obtained fromthe datastore 197. The classifiers 196 may be trained using supervisionfrom the question answerer 192. For example, web pages may be obtainedfrom the corpora 195, and human users may assign annotations to them. Asone example, such users may identify topics that are addressed in thepages. Those answers provided by users may then be provided asinformation to the datastore 197 in much the same manner that answers toquestions posed in the analyzer process would be provided. Such answersmay also be used to annotate the object or item, where the annotationsare provided back to the corpora for storage in association with theunderlying item or object.

The classifier, when properly trained, can also operate in anunsupervised manner. For example, the classifier may obtain one or moredocuments from the corpora 195 and may match such documents to domainsobtained from the datastore 197 to identify attributes and associatedvalues for such documents. As one example, described above and below,the classifiers 196 may obtain a document when a user attempts to uploadit to a system. The classifiers 196 may then match the document invarious known manners to the domains from the datastore 197 to identifyattributes for the document. In addition, the classifiers may attempt todetermine whether an appropriate value is present in the document forthe particular attribute. For example, if an attribute of “car” isidentified, and a color of “red” is also in the document, the classifiermay check against the domains to determine that red is an acceptablevalue for a car and to conclude that the term red in the document ismeant to refer to the term car, i.e., that red is a value of the carattribute.

The classifiers 196 may also determine, in various known manners, alevel of confidence in assigning a particular attribute and value forthe obtained document. If the confidence value is sufficiently high, theattribute and value may simply be applied as annotations for the item ordocument. If there is a middle level of confidence, the proposedannotations may be presented to the question answerer 192 in mannerslike those described above and below, to obtain confirmation of whetherthe attribute and value is accurate for the item or document. If theconfidence value is sufficiently low, the information may simply bepassed by, under the assumption that it cannot be confirmedautomatically, and that an attempt to present the proposed answer to thequestion answerer may lead the person providing the answers to questionthe quality of the system 190. For example, a user may be less likely touse a system in the future if the system asks the user if the color of acar is “oil” or “vinyl.”

FIG. 2 is a flowchart of an example method 200 for performing attributeextraction. The process begins with the receipt of a data object (202).This can occur, for instance, by the server 109 receiving data uploadedfrom a client 115. The uploaded data can include, for example,information to be made available on the Internet such that a web browserdirected to a web server can access the data after it has been uploadedfrom the client 115. The uploading can be accomplished via any knownfile transfer means. In some implementations, the uploaded data objectis accepted by the server 109 from the client 115 via a web uploadinginterface. That is, the server 109 can provide a webpage to a webbrowser on a client 115. The client 115 can display the webpage on adisplay through its web browser, with the client 115 operating as userinterface device. The web page can allow a user of the client 115 toselect a file local to the client 115, such as a file in the memory 160or on a storage device 185 for transfer to the server 115.

Upon receiving the uploaded data object, the server 115 can store thedata object in the data object database 141. The database 141 need notbe local to the server 115. That is, the data object can be stored insome other server connected to the network(s) 110. An extraction processis performed on the data object (204). The extraction process can beperformed by the extraction module 126. The extraction process can varydepending on the type of data object being analyzed. For example, a filecomprising ASCII text can be scanned for extraction targets, where theextraction targets comprise terms found in the ontology database 140,synonyms of those terms, antonyms of those terms, or words have otherrelationships to terms found in the ontology database 140. An audio filecan be passed through a speech-to-text program and the resulting textcan be scanned for extraction targets. An image file or video file canbe passed through one or more image recognition programs and theresulting output can be scanned for extraction targets.

The results of the extraction process can comprise a list of knownontology terms and candidate ontology terms. Known ontology terms areterms which were found in the both the data object and the ontologydatabase 140, and/or terms from the ontology database 140 known to berelated in some manner (by either a relationship stored in the ontologydatabase or a rule used by the extraction module 126) to terms found inthe data object (or in the resulting output from intermediate operationsas described above with respect to non-text files). Candidate ontologyterms are terms found in the data object or terms known to be related insome matter to terms found in the data object that the extraction moduleidentifies as potentially contextually relevant to the data object. Thatis, candidate ontology terms are terms which the extraction processidentifies as possibly valuable in characterizing the data object.

Following the extraction process, the method can receive feedback on theresults of the extraction process (206). Receiving feedback, in someimplementations, can include soliciting feedback by sending a feedbackform web page to a web browser of the client 115 that uploaded the dataobject. The feedback form web page can be generated by the feedbackmodule 125. The feedback form web page can include the extractedontology terms comprising the known ontology terms and/or the candidateontology terms as they were identified by the extraction process. Theform can request information from a user as to whether the extractedknown ontology terms accurately characterize the data object, and/orwhether the candidate ontology terms are contextually relevant to thedata object. The feedback form can, for example, solicit feedback usingweb check boxes, radio buttons, text input fields, and other forms ofweb page input. A submit button or other indication that the form hasbeen completed can be used to transfer the feedback to the server 109.

In some implementations, the information is gathered from the clientuser that uploaded the data object. This user can have knowledge of theuploaded data object beyond that which the extraction process is able todetermine. In some implementations the candidate ontology terms areattributes and/or attribute values that the extraction process hasidentified as possibly valuable. The candidate ontology terms can beidentified, for example, by their proximity to known ontology terms inthe data object (for example, the term's location in a text document,their location in the document with respect to known ontology terms,spatial and relationships in an image and/or video, chronologicalrelationships in audio and/or video, and similar volume level in audioand/or video). The candidate ontology terms can be identified in textdata using known language characteristics for the language used in thedata. For example, if the extraction module encounters a verb followedby a word that it does not recognize which is followed by a noun that isknown in the ontology database as an attribute, the extraction modulecan identify the middle word as a candidate ontology term as a value forthe attribute noun.

Some implementations utilize the extraction results consistent with thefeedback given (block 308). Using the feedback, the system can insurethat the extraction process is accurate as well as train the extractionprocess by learning from extractions resulting in both confirmations andrejections. Candidate ontology terms that are confirmed as beingcontextually relevant to the data object can be added to the ontologydatabase, and the extraction process can also be trained based on thisfeedback.

FIG. 3 is an example text document data object 300. The example textdocument data object 300 can be uploaded to the server 109 from a client115. The text comprises the nursery rhyme “Mary Had a Little Lamb”. Theextraction module 125 can analyze the text document to perform theextraction process.

FIG. 4 shows example extraction results 400 for a text document dataobject. The example extraction results 400 correspond to processing of“Mary Had a Little Lamb”. The extraction results include two lineslisting the attribute “title” with the respective values “Mary had alittle lamb” and “Mary had a little lamb a nursery rhyme”. The attribute“title” can be known in the ontology. The extraction module can, forexample, determine that because this text appears on the first line ofthe document that it is potentially a title. Two title results can bedue to “Mary had a little lamb” being bolded in the first line of thedocument and/or appearing repeatedly in the body of the document.

The term “Mary” can be known in the ontology as a value for theattribute “name” due to it being part of a base ontology (an initialontology before training) or due to an earlier extraction and feedbackprocess. Two “topic” attributes are listed including “education” and“fleece”. The term “education” can be known in the ontology as a valuefor the attribute “topic”, again due to it being part of a base ontologyor a due to an earlier extraction process. The attribute value pair“topic: education” can be returned as part of the results set due to thewords “school”, “teacher” being related to “education” in the ontology.The term “fleece” may not be known in the ontology. It can appear in thelist because the extraction module recognizes it is a noun due to itbeing followed in the document by the words “is white”.

The term “lamb” can be known in the ontology as a type of “animal” aswell as a type of recipe due to earlier recipe submissions used to trainthe extraction process. This can result in the extraction moduleoutputting the potential “document type” of “recipe”. The color “white”can be recognized as well as the adjective “little” related in theontology to the term “small”. The word “snow” can be known in theontology as a “weather condition”. Finally, due to the formatting of thedocument, the reoccurrence of words, and/or the detection of a rhymingscheme, the extraction module can return “document type” attributes of“song lyrics” and “poem”. The extraction results can be sent from theextraction module 126 to the feedback module 125 for creating andpresenting a feedback form to the client 115.

FIG. 5 shows an example feedback form 500. The example feedback form 500can be generated by the feedback module using the extraction results ofFIG. 4. The example form includes example answers as completed by ahypothetical user of the client 115. The form can indicate the URL ofthe uploaded data object. The form can also request feedback based onthe extraction results. The form has been completed to indicate that thetitle is “Mary had a little lamb”, and that it is about someone named“Mary”. The topic of the document is indicated as a nursery rhyme as“education” and “fleece” were either erroneous or suboptimal topics forthe document. The form has instead been completed using a type in fieldto indicate that the topic is a “nursery rhyme”. “Fleece” was not knownin the ontology, and here the form indicates that in this case the term“fleece” is not contextually relevant to the analyzed data object. Theform indicates that the document is about something white, a lamb, andsomething small.

The completed form indicates that the document is not a recipe and thatit is not a recipe for a lamb dish. The feedback module can make one ormore form fields dependent on one another. For example, upon the answerto the question “Is your document a recipe?” the following question “Isthe recipe a lamb dish?” can be completed automatically and/or grayedout such that it can no longer be answered. This behavior can be createdusing, for example, Javascript in the web page created by the feedbackmodule 125. The remaining questions are also completed as the dataobject submitter sees as appropriate. Selection of the “SUBMIT” buttoncan cause the completed feedback form to be sent to the server 115.While “yes” or “no” questions are primarily depicted, the feedbackquestions can be more complex. Modes of form input can also includeradio buttons, select boxes, and the like.

In some implementations, answers to feedback questions can result inadditional questions. The feedback is received by the server 115 andthen utilized at least in part to characterize the data object, trainthe extraction process, and/or update the ontology, if appropriate.Attribute-value pairs confirmed as relevant through the feedback processcan be stored in association with the data object to assist searchengine users in retrieving the data object at a later time. The dataobject can be served to a remote computer through, for example, a webserver interface of the system 109.

FIG. 6 is a flowchart of an example method 600 for processing receivedfeedback. Feedback is received at the server 109 (602). If there is nounprocessed feedback remaining (604), the process ends (606). If thereis unprocessed feedback remaining, the next attribute, its correspondingvalue, and the feedback response are processed (607). If the feedbackconfirms that the attribute-value pair accurately characterizes the dataobject (608), the extraction process is trained based at least in parton the confirmation (610). If the attribute-value pair is not confirmedthe feedback is analyzed to determine if the attribute-value pair isrejected (612). If the attribute-value pair is rejected, the extractionprocess is trained based at least in part on the rejection (614). If theattribute-value pair was neither confirmed nor rejected, processingreturns to block 604. Following the training of the extraction processwhere the attribute-value is confirmed (610), an association can becreated between the data object and the confirmed attribute-value pair.For example, the attribute and value can be stored with the data object.The confirmed attribute-value pair can be analyzed to determine if theattribute is known to the ontology (616). If the attribute is not knownin the ontology it can be added to the ontology (618). The confirmedattribute value pair can be analyzed to determine if the value is knownin the ontology (620). If the value is not known in the ontology it canbe added to the ontology (622). Processing can return to 604 wherefeedback for the next attribute-value pair is processed, or if there isno more unprocessed feedback, the method ends (606).

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a tangible program carrier forexecution by, or to control the operation of, data processing apparatus.The tangible program carrier can be a propagated signal or a computerreadable medium. The propagated signal is an artificially generatedsignal, e.g., a machine generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a computer.The computer readable medium can be a machine readable storage device, amachine readable storage substrate, a memory device, a composition ofmatter affecting a machine readable propagated signal, or a combinationof one or more of them.

The term “data processing apparatus” encompasses all apparatus, devices,and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, to name just a few.

Computer readable media suitable for storing computer programinstructions and data include all forms of non volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter described in thisspecification have been described. Other embodiments are within thescope of the following claims. For example, the actions recited in theclaims can be performed in a different order and still achieve desirableresults. As one example, the processes depicted in the accompanyingfigures do not necessarily require the particular order shown, orsequential order, to achieve desirable results. In certainimplementations, multitasking and parallel processing may beadvantageous.

The invention claimed is:
 1. A method comprising: receiving a documentfrom a user device; analyzing the document to extract a plurality ofvalues having corresponding attributes associated with attribute-valuepairs in one or more domain models, wherein the analysis is based atleast in part on associating the document with the one or more domainmodels of a plurality of domain models generated through analysis of aplurality of documents from one or more corpora and using an ontologyassociated with the one or more domain models to determine one or moreattributes from the document, wherein the ontology includes termsrelevant to particular domains corresponding to the one or more domainmodels, and wherein the attributes from the document are associated withparticular terms of the ontology; associating the determined one or moreattributes with the document; and using the associated attributes togenerate search results responsive to a received search query.
 2. Themethod of claim 1, wherein the analysis is used to extract one or morevalues from the document as corresponding to ontology terms and one ormore candidate ontology terms associated with respective attributes. 3.The method of claim 2, comprising providing a user interface to aclient, the user interface configured to present at least one questionas to whether an extracted value of the one or more values paired withits corresponding attribute accurately characterizes the document,wherein at least one extracted value paired with its correspondingattribute is a candidate ontology term not found in the ontology and ispresented in the at least one question.
 4. The method of claim 3,comprising receiving a response to the at least one question from theclient though the user interface indicating that the attribute-valuepair accurately characterizes the data object.
 5. A system comprising:one or more computers including one or more storage devices and one ormore processor devices configured to perform operations comprising:receiving a document from a user device; analyzing the document toextract a plurality of values having corresponding attributes associatedwith attribute-value pairs in one or more domain models, wherein theanalysis is based at least in part on associating the document with theone or more domain models generated through analysis of a plurality ofdocuments from one or more corpora and using an ontology associated withthe one or more domain models to determine one or more attributes fromthe document, wherein the ontology includes terms relevant to particulardomains corresponding to the one or more domain models, and wherein theattributes from the document are associated with particular terms of theontology; associating the determined one or more attributes with thedocument; and using the associated attributes to generate search resultsresponsive to a received search query.
 6. The system of claim 5, whereinthe analysis is used to extract one or more values from the document ascorresponding to ontology terms and one or more candidate ontology termsassociated with respective attributes.
 7. The system of claim 6,comprising providing a user interface to a client, the user interfaceconfigured to present at least one question as to whether an extractedvalue of the one or more values paired with its corresponding attributeaccurately characterizes the document, wherein at least one extractedvalue paired with its corresponding attribute is a candidate ontologyterm not found in the ontology and is presented in the at least onequestion.
 8. The system of claim 7, comprising receiving a response tothe at least one question from the client though the user interfaceindicating that the attribute-value pair accurately characterizes thedata object.
 9. A method comprising: analyzing a document using one ormore trained classifiers to extract a plurality of text values from thedocument and determine corresponding attributes for the text valuesbased at least in part on an ontology containing a plurality of ontologyterms relevant to a particular area of interest and that are associatedwith particular attributes, wherein each of the corresponding attributesis associated with a respective confidence value determined by one ofthe trained classifiers, wherein the confidence value represents a levelof confidence in assigning the respective attribute to the document; inresponse to a particular text value that is associated with a confidencevalue that does not satisfy a threshold but is greater than a minimumconfidence value, presenting the corresponding attribute as a questionto a client user; assigning each of one or more of the plurality of textvalues to a corresponding attribute for the text value that isassociated with a respective confidence value that satisfies thethreshold; associating the attributes with the document; and using theassociated attributes to generate search results responsive to areceived search query.
 10. The method of claim 9, comprising: providinga user interface to a client, the user interface configured to presentat least one question as to whether an extracted text value of theplurality of text values paired with its assigned attribute accuratelycharacterizes the document.
 11. The method of claim 10, wherein at leastone extracted text value is a candidate ontology term not found in theontology and is presented in the at least one question.
 12. The methodof claim 9, comprising analyzing a plurality of objects from a corporato generate one or more domain models, and wherein analyzing thedocument comprises associating the data object with the domain models.13. A system comprising: one or more computers including one or morestorage devices and one or more processor devices configured to performoperations comprising: analyzing a document using one or more trainedclassifiers to extract a plurality of text values from the document anddetermine corresponding attributes for the text values based at least inpart on an ontology, wherein each of the corresponding attributes isassociated with a respective confidence value determined by one of thetrained classifiers; in response to a particular text value that isassociated with a confidence value that does not satisfy a threshold butis greater than a minimum confidence value, presenting the correspondingattribute as a question to a client user; assigning each of one or moreof the plurality of text values to a corresponding attribute for thetext value that is associated with a respective confidence value thatsatisfies the threshold; associating the attributes with the document;and using the associated attributes to generate search resultsresponsive to a received search query.
 14. The system of claim 13,comprising: providing a user interface to a client, the user interfaceconfigured to present at least one question as to whether an extractedtext value of the plurality of text values paired with its assignedattribute accurately characterizes the document.
 15. The system of claim13, wherein at least one extracted text value is a candidate ontologyterm not found in the ontology and is presented in the at least onequestion.
 16. The system of claim 13, comprising analyzing a pluralityof objects from a corpora to generate one or more domain models, andwherein analyzing the document comprises associating the data objectwith the domain models.